1. Field of the Invention
This invention relates to software testing. More particularly this invention relates to model based automatic generation of test programs for the validation of middleware.
2. Description of the Related Art
Middleware is software that functions as a conversion or translation layer between applications or between an application and the operating system. Middleware solutions have been developed in order to enable applications to communicate with other applications. The applications may execute on different platforms, be produced by different vendors or both. Today, there is a diverse group of products that offer packaged middleware solutions. One of the characteristics of middleware is that its software is “programmable”. In other words, the user can program the middleware to behave in a number of different ways.
Traditional software testing involves study of the software system by the tester, who then writes and executes individual test scenarios that exercise the software being tested. In the case of middleware, testing typically requires validation of many functional levels, which increases the complexity of the task. Test software for a middleware system is itself an application which needs to be tested.
More recent model-based approaches to test generation have common problems that this invention builds upon. In each case, the number of unique paths, or generated test programs is an exponential function of the number of modeled states and transitions. Thus, as the scope of the modeled behavior grows, the time to exhaustively generate test cases, and more significantly, the time needed to execute the generated test cases grows exponentially. This growth places a practical limit on the complexity of the program behavior to which automated model-based test generation can be applied. The invention focuses and therefore reduces the number of tests to a practical level. In so doing, the invention raises the practical limit on the complexity of the software program to which automated model-based test generation may be applied.
Conventional testing of established types of middleware involves the use of benchmark sets of input programs. These programs are seldom run with real input data. Instead, the output of the middleware, for example a compiler, is compared with the results of other compilers for the same input code. This testing methodology is inappropriate for new products, for which there are no benchmarks, and for which there is no comparable middleware to evaluate the test.
Typical of conventional approaches for generating test programs from a model is U.S. Pat. No. 5,394,347 to Kita et al. which discloses a method of modeling a specification as an extended finite state machine, then performing a depth-first traversal of the resulting state diagram to generate a path file as a basis for a test program.
In U.S. Pat. No. 5,918,037 to Tremblay et al., it is proposed to employ a test generator that automatically produces test programs based on a finite state machine model of the software. Limiting the number of test programs is achieved by controlling loop execution, and by appropriately setting the coverage level for the model, known as “transition cover testing”. This approach seeks to specify during the test program generation process that each transition within the finite state machine model be exercised once. The generator is capable of specifying different coverage levels for selected portions of the program under test, so that critical portions might be exhaustively tested, while other portions receive less comprehensive testing.
During the past decade, model-based random test program generators have become popular in processor architectural design verification and software testing. An example of such a random test generators include the IBM tool, “Genesys”, which is disclosed in the document Model-Based Test Generation for Process Design Verification, Y. Lichtenstein et al., Sixth Innovative Applications of Artificial Intelligence Conference, August 1994, pp. 83–94.
Another conventional test generator, AVPGEN, is disclosed in the document AVPGEN—A Generator for Architecture Verification Test Cases, A. Chandra, et al. IEEE Trans. Very Large Scale Integration (VLSI) Syst. 3, No. 2, 188–200 (June 1995).
None of the techniques disclosed in the above noted documents is well suited for solving the particular issues presented by middleware.